Evaluating the Operating Characteristics of a Class of Closed Multiple Comparisons Procedures: A General Framework

Analytic formulas are developed for various types of power and error rates of some closed testing procedures. The formulas involve non-convex regions that may be integrated with high, pre-specified accuracy using available software. The non-convex regions are represented as a union of hyper-rectangles. These regions are transformed to the unit hypercube, then summed, to create an expression for power that is the integral of a function defined on the unit hypercube. This function is then evaluated using existing quasi-Monte Carlo methods that are known for their accuracy. Applications include individual, average, complete, and minimal power, for closed pairwise comparisons (max T-based), as well as fixed sequence and non-pairwise comparisons. An extension to Bayesian predictive power also is given. The methodology extends naturally to the evaluation of combined directional and nondirectional Type I error rates, which we investigate thoroughly and find no evidence of excesses in applications involving the noncentral multivariate t distribution.